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Description 

BACKGROUND OF THE INVENTION 

s This invention relates to a neural network circuit for executing innage recognition processing. 

Much Interest has recently been shown in a field of neutral networl< in data processing. The neural network is 
brought up from simulatiori of a neuron structure of a brain of a living thing. Many neural networks are accomplished 
by a conventional von Neumann sequential computer whose processing speed is extremely low. Therefore, the neural 
network Is now tried to be structured by exclusive electric circuits. There are various kinds of neural network structured 

10 by exclusive electric circuits, such as a multi-layered neural network. FIG.8 shows the multi-layered heural network 
which has neurons having branch ability and integration ability and is provided with an input layer, an output layer and 
an intermediate layer of multiple layers interposed therebetween. 

The multi-layered neural network shovyn In FIG.8 has three layers of: an input layer composed of two neuron 
elements 111 , 112 to which input vectors 1=1 , \=2 are respectively inputted, an output layer composed of two neuron 

« elements 330, 340 respectively regarding outputs o=1 , p=2, and an intermediate layer composed of six neuron elements 

121 - 124, 131, 132 formed in two layers. The intermediate layer is disposed between the input layer and the output 
layer. Between neuron elements in the respective layers, synapse groups 41 - 43 are disposed for setting coupling 
load therebetween. Each coupling load of synapses of the synapse groups 41 - 43 is changeable by learning. 

A neural network agreeing to a recognition object is structured by learning each coupling load of synapses of the 

20 synapse groups 41 - 43i changing each coupling load sequentially adequately As a leaming method of each coupling 
load of synapses of the synapse groups 41 - 43, a back propagation method (BP method) is generally known in the 
art, which is very time consuming for learning and additional leaming and whose learning algorithm is unsuitable for 
constructing the hardware. . 

A neural network circuit shown In FIG.9 is proposed which is capable of high speed initial learning and additional 

25 learning and whose algorithm is suitable for constructing the hardware. 

The neural network circuit in FIG.9 is a neutral .network developing the network structure in FIG.6 into a tree-like 
branch structure, and is a three-layered neural network provided with an input layer, an intermediate layer and an 
output layer. The input layer is composed of neuron elements 11-11. 12-11 for only branch operation to which input 
vectors i=1 , i=2 are respectively inputted. The intermediate layer is composed of 24 neuron elements 11-21 - 11-24, 

30 1 1 -31 - 1 1 -38, 1 2-21 - 1 2-24, 1 2-31 - 1 2-38 for only branch operation which are formed in two layers, and has networks 
1 3-1 , 1 3-2 in tree-like branch structure in number of input vectors 1=1 , i=2 of the input layer (i.e.. two). The output layer 
is composed of two output neuron elements (output units) 310. 320 for only Integration operation, which respectively 
regard outputs o=1, o=2, and sums outputs from the upper 16 neuron elements 11-31 - 11-38, 12-31 - 12-38 of the 
Intermediate layer. Between the intermediate layer and the output layer a synapse group 4 is disposed for setting 

35 respective coupling loads between the neuron elements. The coupling loads of each synapse of the synapse group 4 
are changeable by leaming. In FIG.9, paths through 12-11 - 12-22 - 12-34 - 310 corresponds to paths through 112 - 

122 - 132 - 330 in FIG.8. Wherein, each coupling load of synapse between the neuron elements 11-11 - 11-38 and 
each coupling load of synapse between the neuron elements 12-11 - 12-38 are not learned and are set necessarily 
according to a value of the input vector inputted into the respective neuron elements 11-11 , 12-11 of the input layer. 

40 As an example of network system which depends on only the value of the input vector and sets necessarily without 

learning, the coupling load of synapse in three-like branch structure, such as shown in FIG.9. there is a network system 
called quiantizer neuron which is disclosed" in "Character Recognition System Using Network Comprising Layers By 
Function", Preliminary iVIaterial for Image Electronic Communication Society's National Conference 1990, pages 77 - 
80 and "Multi-Functional Layered NetWork using Quantizer Neurons", Computer World '90. November. In this kind of 

4S network structure, the coupling loads of synapses of a final layer Is changed independent form other synapses, which 
leads to high speed initial learning and additional learning and makes the learning algorithm suitable for constructing 
the hardware. 

In the recognition method in the network system shown in FIG.9. output values of the two neuron elements 310, 
320 for only integration operation which are provided at an output layer are judged as to which is the largest and the 
so address of the neuron element whose output value is the largest is made a recognition result. In the integration method 
in the neuron elements 310. 320 of the final layer for integrating outputs of the neuron elements 11-31 - 11-38. 12-31 
- 12-38 of the intermediate layer, the respective output values of the iriterrnedlate layer and the respective coupling 
loads set in the synapse group 4 are multiplied and summed for integration. 

The integration method in neuron elements are explained, with reference to FIG. 10. 
ss In FIG. 10, the output neuron elements 310, 320 and the synapse group 4 are identical with those in FiG.9. Ref- 

erences f1 and f2 denote intermediate output values of the neuron elements 11-31, 11-32 in FIG.9 respectively In 
accordance with the above-mentioned references, the neuron elements 11-31 - 12-38 are branch points of an input 
signal, so that an output value from the neuron element 11-31 to the output neuron element 31 0 and an output value 
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from the neuron element 11-31 to the output neuron element 320 are equal to each other and are indicated by f1. 
Respective coupling load calculations of synapses to the output neuron elements are executed by respective coupling 
calculation executing parts 4-11 - 4-22. The coupling load calculation executing part 4-11 Is composed of a multiplier 
1 1,2 and a coupling load coefficient W1 1 which is multiplied with the output value f I corresponding to the intermediate 
s layer to output a hiultiplled. result. The coupling load calculation executing parts 4-12 - 4-22 have the same function as 
the coupling load calculation executing part 4-11 , and have a different coupling load coefficient frorri one another. The 
integration calculations in the output neuron elements 310, 320 are expressed in following respective equations. The 
thus Integrated output V3lues of the neuron elements 31.0, 320 are judged as to which Is the largest, and an address 
of the neuron element whose output value is. the largest is made a recognition result. 

10 

output of output neuron element 310 = W1 1 X f1+W12 X f2+„. 



output of output neuron element 320 = W21 Xf1+W22xf2+..; 

The learning algorithm In network system shown In FIG.9 uses the learning rule of Hebb, in which if the recognition 
result is false, the coupling load of the synapse group 4 to an output neuron element to be true is fortified until the 
value of the output neuron element to be true Is the largest. output value by a supervisor input in FIG.9. As to the 
20 fortifying method, the coupling load coefficient is added according to the output value of the neuron elements 11 -31 - 
11-38, 12-31 - 12-38. 

The fortifying method of coupling load of the synapse is explained, with reference to FIG. 11 . 
FIG.II shows the coupling load W11 of FiG. 10 in enlarged scale. The Interrtiediate layer output f1 isaddedtothe 
present coupling load W11 according to a leaming signal. The change in coupling load by leaming is expressed as 
S6 follows: 

W11 =W11 +fi ■ 

30 In the multi-layered neural network structure which has the intermediate layer of tree-like branch structure, executes 

integration of synapses by output neuron elements of the final layer and executes leaming by changing the coupling 
loads of the synapses of the final layer, the coupling load change is executed independent from the other synapses, 
which leads tahigh speed initial learning and additional learning and makes the leaming algorithm suitable for con- 
structing the hardware. 

35 Recognition accuracy in the above multi-layered neural network of tree-like brunch structure is, however, low in 

case where anew unlearned data is recognized after an initial learning. The inventors have studied the reasons and 
consider the following is one of the reasons: in case where some kinds of input data are all identified by learning in the 
above neural network and variance of one kind of input data is small, a coupling load of the synapse group to the output 
neuron elements for recognizing a data similar to the input data is inflated at learning of the similar data so as to clarify 

40 a difference between the similar data and the one-kind Input data, with a result that the similar data Is mis-recognized 
as an output result of output neuron element having as an input the Inflated coupling load at next recognition of an 
unleamed data different from the similar data under such a condition since the inflated coupling load is extremely large 
compared with the other coupling loads of the unlearned data when the synapse group of the Inflated coupling load is 
Included in the synapse group to the output neuron element for the unlearned data recognition. 

45 At the initial learning, the coupling load of synapse to the output neuron element is gradually increased according 

: to the numberof times at learning, which requires bit accuracy (bit word length) of coupling load and increases hardware 
in size whteh Is required for coupling load memory. The above-mentioned references disclose that the coupling load 
memory of about 16 bits Is'required for 1 3-font learning of 62 kinds of character data according to a data group to be 
recognized. This means a large-sized hardware required for the coupling load memory. 

so 

SUMMARY OF THE INVENTION 

The object of the invention is providing a neural netvrork circuit capable of solving the problems In the neural 
network of tree-like branch structure, of improving a recognition accuracy for unlearned data and of reducing the amount 
55, of hardware required for coupling load memory. 

To attain the above object. In the multilayered neural network of the present invention, connection of the synapses 
to the output neuron elements is controlled by leaming, thus differing from the conventional neural network circuits 
wherein the coupling load of synapses to the output neuron elements Is calculated by means of weights. 
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A multi-layered neural network circuit according to the present Invention is provided as set out in claims 1 and 5. 

In a further embodiment of the neural network circuit, each output of the Intermediate layer has two kinds of values 
of "HIGH" and "LOW", and the respective output units count the number of "HIGH"s among the outputs of the inter- 
mediate layer connected by the respective connection control circuits. 
s According to the neural network circuit with the above construction, the number of learning times of synapses to 

the output neuron elements Is memorized In the learning-time memory, and only paths between the outputs whose 
numbers of learning times (number of times that non-zero values are outputted or number of times that each of the 
outputs exceeds the set value In each output of the neuron elements within the intermediate layer of network In tree- 
like branch structure) exceed the set threshold among outputs of the neuron elements within the Intermediate layer 
10 and the respective output neuron elements are connected by the connection control circuit only when the number of 
learning times exceeds the set threshold. As a result, each output neuron element executes summation of all outputs 
of the neuron elements within the intermediated layer to recognize the Input data. The Input data recognition depends 
on the path connection between the outputs whose numbers of learning times exceed the set threshold among the 
outputs of the neuron elements within the Intermediate layer and the respective output neuron elements. Since no 
IS weight are used In the connected paths, the local inflatton In coupling load of synapse as the conventional one does 
not occur, thus enhancing the recognition accuracy tor unlearned data. 

By clipping the number of learning times In the leamlng-tlme memory to the upper limit value, the hardware size 
required for memorizing the nunriber of learning times is reduced. 

lyiany paths among paths between outputs of neuron elements within the intermediate layer and the output units 
20 . are connected unnecessarily owing to excessive learning. However, the unnecessary paths are disconnected so as 
to connect only the necessary paths by subtracting the set value from each number of learning times in all learning- 
time memories or by Increasing the threshold of the threshold processingcircuit by the set value. Thus, noise component 
of data to.be recognized at excessive learning Is reduced, while further enhancing the recognition accuracy for un- 
learned data. 

25 Moreover, in one embodiment of the neural network circuit of the present Invention, instead of leamlng-tlme mem- 

ory, the flag memory Is provided for memorizing the presence of learning history erf synapses to the output neuron 
elements, which further reduces the amount of hardware. 

Instead of summation of output values of the intermediate layer being achieved by the respective output neural 
elements of the output layer, the number of times that outputs of non-zero value among outputs of neuron elements 
30 within the intermediate layer are outputted are counted, which further reduces the hardware amount of the integration 
circuit including the output layer 

Other objects and features of the neural network circuit of the present Invehtlon will appear more fully from the 
following description with accompanying drawings. 

35 BRIEFDESCRIPTIONOF THE ACCOMPANYING DRAWINGS 

Accompanying drawings Show preferable embodiments of the present invention, in which: 

FIG.1 is a diagram showing a construction of a neural network circuit in a first embodiment; 
40 FIG.2 is a diagram of a learning calculation circuit of a leamlng-tlme memory; 

FIG.3 is a diagram of a leaning calculation circuit showing a first modeled example of learning-time memory; 

FIG.4 is a diagram of a learning calculation circuit showing a second modified, example of learning-time memory; 

FIG.5 is a diagram showing a construction of a neural network circuit In a second embodiment; 

FIG.6 is a diagram showing a construction of a neural network circuit in a third embodiment; 
45 FIG. 7 is a diagram showing a construction of a neural network circuit in a fourth embodiment; 

FIG. 8 is a diagram showing a construction of a conventional multi-layered network circuit; 

FIG. 9 is a diagram showing a construction of a conventional multi-layered network circuit in tree-like branch struc- 
ture; 

FIG. 10 is a diagram for explaining an integration method of Integral neuron elements In the conventional rnultl- 
50 layered neural network circuit In tree-like branch structure; and 

FIG.11 isadlagram of a conventional coupling load calculation circuit of a coupling load memory. 

. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

55 Description is made below about preferred embodiments of the present invention, with reference to accompanying 

drawings. 

FIG.1 shows a construction of a neural network circuit, and corresponds to the conventional example in FIG.1 0. 
As far as Is possible the same reference numerals have been used as in FIG. 10. 
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In FIG.1, referenciB numerals 310 and 320 are output neuron elements of a final layer for Integrating intermediate 
layer outputs of neuron elements 11-31 - 11-38, 12-31 - 12-38 in FIG.9. Referencesfl andf2 are, as mentioned in the 
conventional example, intermediate layer output values of neuron elements 11-31 and 11-32 in FIG.9 respectively. In 
accordance with the above mentioned references, the neuron elements 11-31 - 12-38 are branch points of an input 
s signal, so that an Output value from the neuron element 11 -31 to the output neuron element 31 0 and an output value 
from the neuron element 1 1 -31 to the output neuron element 320 are isqual to each other and are indicated by f I. The 
calculation of connection of synapses to the output neuron elements is executed by connection calculation executing 
parts 4-11 -4-22. 

The connection calculation executing part 4-11 is composed of a learning-time memory 113, a threshold processing 
10 circu it 1 1 4 for th reshold-processing the learning-time memory 1 1 3, namely for comparing the number of learning times 
stored in the corresponding learning-time memory 113 with a set threshold and for outputting a first set value when 
the number of learning times of the corresponding learning-time memory is equal to or more than the threshold and 
outputting a second set value when the number of times at learning of the corresponding learning-time memory is less 
than the threshold, and a connection control circuit 115 controlled by two kinds of control signals from the threshold 
1S processing circuit 114. The connection control circuit 115 controls connection synapses between the neuron elements 
within the Intermediate layer and the output neuron elements. The connection calculation executing parts 4-12 - 4-22 
have the same function as of the connection calculation executing part 4-11 , and have a different number of times at 
learning from one another, if the threshold of the threshold processing circuit 114 is 1 , the Intermediate layer output f1 
is outputted when the learning-time value R11 (the number of times at teaming) of the leaming-tlme memory 113 in 
so the connection calculation executing part 4-11 is 1 or more and 0 is outputted without connection of the intermediate 
layer output f1 when the leiaming-time value R11 of the learning-time memory 113 is 0. In the output neuron elements 
31 0, 320, only the intermediate layer outputs of connected synapses are added for integration among the intermediate 
layer outputs. 

The thus Integrated output values of the output nisuron elements 31 0, 320 are judged as to which is the largest so 
2S as to make an address of the output neuron elertient whose output yalue Is the largest a recognition result. 
Learning algorithm in the network systern shown in FIG.T Is discussed next. 

First, all learning-time memories of synapses continuing to the output neuron elements are set 0. Then, an initial 
learning is executed only one time to all data to be initial-learned. The learning method is as follows: the values in all 
learning-time memories of synapses whose intermediate layer outputs are not 0 are incremented by a supervisor input 
30. in F1G.1 through connection synapses connected to the output neuron elements corresponding to the input data. It 
may be possible to increment the values in the leaming-tlme memories of synapses whose intermediate layer outputs 
are equal to or rrx>re than a threshold. ^ 

FIG.2 shows an example of an updater of the learning-time memory 113 in FIG.1 in enlarged scale. The present 
learning-time value R11 is incremented by 1 according to a learning sigrial. Change In number of learning times by 
3S learning is expressed by the following equation; 

R11 =R11 + 1 

40 As described above, inthisembodiment.thenumberof learningtimesof asynapseconnectedtotheoutputneuron 

elements is memorized, and only the outputs of neuron elements within the intermedete layer from synapses whose 
numbers of learning times exceed the threshold are summed, Thus, the local inflation in coupling load of synapse due 
to small variance of one kind of input data in additton to similar data is prevented and the recognition accuracy for 
unlearned data is enhanced. 

4S FIG,3 shows a modified example of a learning calculation circuit of the learning-time memory 113. As shown in 

FIG.3, an upper limiter 113 as upper limit clipping means is provided on the input side of the learning-time memory 
11 3. This reduces a memory capacity of the learning-time memory 1 1 3. Different from the conventional construction 
wherein the coupling load of the synapses to the output neuron elements is calculated by rneans of weights. In the 
present invention, the connectfon synapses to the output neuron elements Is calculated according to the number of 

so learning times. 

When a limit value (upper limit of the number of learning times) of the upper Iimiter113a shown in FIG.3 Is set to 
3, only two bits are required for memorizing the number of learning times, which is one eighth of 16-bit coupling load 
memory in the conventional one. Further, the recognition accuracy is increased to about 86% (63% In the conventional 
one) in a recognition test for unlearned data after the Initial learning according to the present invention with the neural 
ss network construction in the above mentioned references. In the recognition test, 62 kinds of 1 3-font character data are 
learned at the initial learning and the recognition accuracy for 62 kinds of 10-font unlearned characterdata is calculated. 

According to the present invention, the recognition accuracy for unlearned data is excellent even with less memory 
capacity required for learning. 
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According to the neural network circuit with the above construction, one-time learning to one input data makes the 
value of the corresponding output neuron element the largest for the same input data without exception, thus enabling 
the initial learning without recognition result with no conditions. Accordingly, the convergence time at the Initial learning 
in the neural network construction In the present Invention Is about one sixtieth to one thousandth of that In the con- 
s ventlonal one. 

While the learning-time memory 113 in FIG.3 has an excellent feature, a problem arises that the recognition ac- 
curacy for unlearned data is lowered owing to excessive learning. Because, the synapses to the output neuron elements 
are unnecessarily connected, receiving many noise components by excessive learning. However, the lowering of the 
recognition accuracy for unlearned data due to excessive learning is prevented by regarding the synapses which are 

JO not so learned in past leaming as noise components and Ignoring the learning history. 

FIG, 4 shows an example of an updater of the learning-time memory which solves the problem which is another 
modification of the leamlng-tlme memory 113 In FIG.1. The learning-time memory in FIG. 4 includes a selector 113b 
as a learning-time updater having a function of decrementing by one the present learning-time value R11 according 
to a learning signal, In addition to the Incrementing function shown In FIG.2. The selector 1 1 3b receives a control signal 

15 for subtraction. When the control signal for subtraction Is Inputted, the selector 1 1 3b outputs -1 to an adder 1 1 3c, halting 
the incrementing function, to decrement by 1 the present learnirig-tlme value R11 . Further, the learning-time memory 
includes a lower limiter 1 1 3d as lower limit clipping means which has a function of limiting a value less than 0 to 0. 

When the recognition accuracy for unlearned data Is lowered due to excessive leaming, 1 is subtracted from all 
learning-time values stored in the learning-time menwrles of the synapses to the output neuron elements according 

20 to the control signal for subtraction. Wherein, the learning-time value R11 less than 0 Is limited to 0 by the lower limiter 
11 3d. With the above function added, the lowering of the recognition accuracy for unlearned data due to excessive 
learning Is prevented. 

As to means for preventing the lowering of the recognltidn accuracy for unlearned data due to excessive learning, 
the same effect can be obtained by adding 1 to the threshold of the threshold processing circuit 11 4 In FIG.1 , besides 

25 the method showing in FIG.4. - 

The updater of the leamlng-tlme memory subtracts 1 In this embodiment, but may execute division If It has the 
function of decreasing the number of times at leaming. 

FIG. 5 shows a construction of a neural network circuit in a second embodiment, and corresponds to FIG.1 0 of the 
conventional example, so the same reference numerals as in FIG.IO have been used for the same elements in FIG. 5. 

30 In FIG.5, reference numerals 310 and 320 are output neuron elements of thafinallayerfor integrating intermediate 

layer outputs of the neuron elements 11-31 - 11-38, 12-31. - 12-38. References fl and f2 are, as mentioned in the 
conventional example, the intermediate layer output values of the neuron elements 11-31 and 11-32 In FIG.9 respec- 
tively In accordance with the above, mentioned references, the neuron elements 11-31 - 12-38 are branch points of 
an Input signal, so that an output value from the neurori element 11-31 to the output neuron element 31 0 and an output 

35 value from the neuron element 11-31 to the output neuron element 320 are equal to each other and are Indicated by 
fl. The calculation of connection synapses to the respective output neuron elements Is executed by the connection 
calculation executing parts 4-11 - 4-22. 

The connection calculation executing part 4-11 is composed of a flag memory 116 and a connection control circuit 
115 to be controlled by the flag memory 116 for controlling connection synapses between neuron elements within the 

40 intermediate layer and the output neuron elements. The connection calculation executing parts 4-1 2 - 4-22 have the 
same function as that of the connection calculation executing part 4-1 1 , and have a different flag value from one another. 
The Intermediate layer output fl Is outputted when a value Qll of the flag memory 116 in the connection calculation 
executing part 4-11 isl and 0 is outputted without connection of the Intermediate layer output f.1 when the value Q11 
of flag memory 116 is 0. In the output neuron elements 310, 320, only the Intermediate layer outputs of connected 

45 synapses are added for integration among the intermediate layer outputs. The thus integiated output values of the 
neuron elements 31 0, 320 are judged as to which is the largest so as to make an address of the output neuron element 
whose output value is the largest a recognition result. 

The learning algorithm in the network system shown In FIG.5 is discussed next. 

First, all flag memories of synapses connected to the output neuron elements are set to 0. Suppose that the outputs 
so of the neuron elements within Intermediate layer output are not connected to the output neuron elements when the 
flag memory Is 0 and are connected thereto when the flag memory Is 1 . Then, at the Initial learning, leaming Is executed 
only one time to all of the data to be initial-learned. The learning method Is that: 1 Is set by a supervisor Input In FIG: 
5 to all values In the flag memories of synapses whose Intermediate layer outputs are not 0 among the synapses 
connected to the output neuron elements corresponding to the input data. Setting to 1 may be conducted to the flag 
ss memories whose intermediate layer output is equal to or more than a set threshold, instead of non-zero intermediate- 
layer value. 

As described above, In this embodiment, the flag memory requires only one-bit memory capacity for one synapse, ' 
which means further reduction of memory capacity than In the first embodiment. Since the threshold processing circuit 
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for the learning-time memory is unnecessary, the size of tfie hardware is expected to reduce. As to the recognition 
accuracy for unleamed data, the equivalent performance is obtained as in the neural network circuit in the first em- 
bodiment. 

According to the neural network circuit of the present invention, the recognition accuracy for unleamed data is. 
s excellent even with less memory capacity of the flag memory which is required for learning. 

Further, in the neural network circuit with the above construction, one-time learning to one input data makes the 
value of the corresponding output neuron element the largest.for the same input data without exception, which enables 
the initial learning without recognition result with no conditioris. Accordingly, the convergence time at the initial learning 
in the neural network construction of the present invention is about one sixtieth to one thousandth of that in the con- 
10 ventional one. 

In the neural network circuit in this embodiment, the recognition accuracy for unlearned datg is lowered because 
of excessive learning. However, such the low recognition accuracy can be ignored in cases of initial learning not to be 
excessive learning, of teaming of data with less noise component or of use of the flag memory in ROM construction 
as a recognition device. 

'S Fl G.6 shows a neural network circuit according to a third embodinnent, and corresponds to Fl G.5 of the conventional 

example, so the same reference numerals as in FIG.5 have been used for the samd elements in FIG.6. 

In FIG.6, reference numerals 310 and 320 are the output neuron elements of the final layer for integrating inter- 
mediate layer outputs of the neuron elements 11-31 - 11-38, 12-31 - 12-38. References fl and f2 are, as described in 
the conventional example, the intermediate layer output values of the neuron elements 11-31 and 11-32 in FIG. 9 re- 

20 spectively. In accordance with the above mentioned references, the neuron elements 11-31 - 12-38 are the branch 
points of an input signal, so that an the output value from the neuron element 11-31 to the output neuron element 310 
and an output value from the neuron element 11-31 to the output neuron element 320 are equal to each other and are 
indicated by f1. Wherein, in FIG.6, there are two kinds, i.e. 1 and 0, of intermediate layer output values fl, f2. The 
calpulation of the connection synapses to the respective output neuron elements is executed by the connection calcu- 

25 lation executing parts 4-1.1 - 4-22. 

The connection calculation executing part 4-11 is composed of the flag memory 116 and a connection control 
circuit 117 to be controlled by the flag memory 116 for controlling the connection of synapse between the Intermediate 
layer and the output neuron elements. The connection calculation executing parts 4-12-4-22 have the same function 
as the connection calculation executing part 4-11 , and have a different flag value from one another. Since the connection 

30 control circuit 117 receives two kinds of input values from the intermediate layer, a logical product circuit, instead of 
the selector shown in FIC3.5, can servo as the connection control circuit 117. In the output neuron elements 310, 320, 
the number of ones of outputs of neuron elements within the interniediate layer of connected synapses to neuron 
elements of the output layer are counted for integration. 

The thus integrated output values of the output neuron elements 310, 320 are judged as to which is the largest so 

35 as to make an address of the output neuron element whose output value is the largest a recognition result. 
The learning algorithm in the network system shown in FIG.6 is discussed next. 

First, all flag memories of synapses continuing to the output neuron elements are set to 0. Suppose that outputs 
of neuron elements within the intermediate layer are not connected to the output neuron elements when the flag memory 
is 0 and- is connected thereto when the flag memory is 1. Then, at the initial learning, learning is executed only one 
40 time to all of the data to be initial-leamed. The learning method is that: 1 is set by a supervisor input in FIG,6 to all 
values in the flag memories of synapses whose intermediate layer outputs are 1 among the synapses connected to 
the output neuron elements corresponding to the input data. 

By employing the neural network circuit of the invention in the second embodiment and in this embodiment, the 
connection control circuit 117 of the synapse to the output neuron elements is simplified and the integration processing 
4S which is executed in the output neuron elements 310, 320 is executed by counting the number of inputted values of 
1 , thus reducing the hardware size, compared with the circuit shown in FIG.5. 

Upon a test of recognition accuracy for unlearned data, about 2 - 3% lowering of recognition accuracy is caused 
compared with the circuits in first and second embodiments. However, the recognition accuracy is much higher than 
that in the conventional learning method, which means applicable into practice depending on a kind of data to be 
so recognized. 

FIG.7 shows a neural network circuit according to a fourth embodiment of the present invention and corresponds 
to FIG.1 , so the sarine reference numerals as in FIG.1 have been used for the same elements in FIG.7. 

In FIG.7, reference numerals 310 and 320 are output neuron elements of the final layer for integrating the inter- 
mediate layer outputs of the neuron elements 11-3f - 11-38, 12-31 - 12-38 shown in F1G.9. References fl and f2 are, 
'55 as mentioned in the conventional example, the intermediate layer output values of the neuron elements 11-31 and 
11-32 in FIG.9 respectively. In accordance with the above mentioned references, the neuron elements 11-31 - 12-38 
are the branch points of an input signal, so that an output value from the neuron element 11-32 to the output neuron 
element 31 0 and an output value from the neuron element 1 1 -32 to the output neuron element 320 are equal to each 
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other and are indicated by f1 . Wlierein, in FIG.7, the intermediate layer output values f 1 , f2 have two kinds of outputs, 
i.e. 1 and 0. The calculation of connection of the synapse to the output neuron elements is executed by the connection 
calculation executing parts 4-11 -4-22. 

The connection calculation executing part 4-11 is composed of the leaniing-time memory 113, the threshold 

5 processing part 114 for threshold-processing the learning-time memory 113 and the connection control circuit 117 to 
be controlled by the control signal of two kirids of values by the threshold processing circuit 114 for contrplling the 
connection synapses between the neuron elements within intermediate layer and the output neuron elements.. The 
connection calculation executing parts 4-12 - 4-22 have the same function as that of the connection calculation exe- 
cuting part 4-11, and have a different learning-time value from one another. Since the connection control circuit 117 

10 receives two kinds of values from the interrhediate layer, a logical product circuit, instead of the selector shown in FIG. 
1 , can serve as the connection control circuit 117. In the output neuron elements 310, 320, the number of outputs 
having a value of 1 of the neuron elements within the intermediate layer of the connection synapses is counted for 
integration. 

The thus integrated output values of the neuron elements 310, 320 are judged as to which is frie largest so as to 
IS make art address of the output neuron element whose output value is the largest a recognition result. 

According to the circuit shown in FIG.6, the connection control circuit 117 of the synapse to the output neuron 
elements is simplified and the integration processing executed in the output neuron elements 310, 320 Is executed by 
counting the number of Input values of 1 , thus reducing the hardware size, compared with the circuit shown in FIG.1 . 
Similar to the circuit shown in FIG.6, the recognition result for the unleamed data Is about 2 - 3% lower than that 
20 in the first embodiment shown in FIG.1 and that in the second embodiment shown in FIG.5. However, the recognition 
accuracy thereof is much higherthan that in the conventional learning method and the circuit is applicable into practice 
depending on a kind of data to be recognized. The circuit shown in FIG.7 can prevent the lowering of the recognition 
accuracy for unlearned data due to excessive learning, as well as the.circuit in FIG.1 . 

In the above embodiments, each connection calculation executing part 4-11 - 4-22 requires a memory for memo- 
es rizing a different learning value. However, the processing in each connection calculation executing part and the Inte- 
gration processing in the output neuron elements can be executed by using one or plural processing devices, sequen- 
tially exchanging the learning memories. Moreover, as described in this embodiment, the connection calculation exe- 
cuting parts may be provided at all synapses to the respective output neuron elements to execute parallel processing. 
The two output neuron elements are discussed for the sake of simplicity, but the neural network circuit of the 
30 present invention is not limited to this example, and may have another number of outputs. 



Claims 

3S 1. A multi-layered neural network circuit provided with an input layer (11-11 - 12-11) to which one or plural Input 
vectors are Inputted, an intermediate layer (11-31 - 11-38,12-31 - 12-38) having neuron elements connected in 
tree-like structure whose outputs are necessarily determined by values of the Input vectors and whose number 
corresponds to the number of the Input vectors of the input layer, and an output layer having one or plural output 
neuron elements (310,320) for integrating all output values of neuron elements within the intermediate layer, the 

40 calculation of connection of synapses to the output neuron elements (310, 320) being executed by connection 

calculating executing parts (4-11 - 4-22), said connection calculating executing parts comprising: 

leaming-time memorie (113), respectively provided so as to correspond to respective paths vyhich connect 
respective outputs of neurons located within the intermediate layer with the respective output neuron elements, 
45 forrespectiveiy memorizing the number of learning tirnes or number of times that an output value of the neuron 

elements within the Intermediate layer is outputted to a corresponding path to the output neuron elements is 

equal to or more than a set value; 

threshold processing circuits (11 4), respectively provided so as to compare the number of learning times stored 
at said learning-time memories with a set threshold value, for outputting a first set value when a number of 
50 ieaming times which is stored in the corresponding learning-time memory is equal to or more than said set 

threshold and outputting a second set value when a number, of times at learning which is stored in the corre- 
sponding learning-time memory is less than said set threshold; and 

connection control circuits (115) for respectively connecting the respective paths between the respective out- 
puts of the neuron elements within the Intermediate layer and the respective output neuron elements when 
55 an output of the corresponding threshold processing circuits (114) is said first set value and disconnecting the 

respective paths therebetween when an output of the corresponding threshold processing circuits (1 1 4) is said 
second set value. 

wherein the respective output units sum the output values of the intermediate layer connected by the respective 
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connection control circuits. 

2. The neural networl< circuit according to Claim 1 , further comprising upper limit clipping means (11 3a) for respec- 
tively clipping the number of teaming times stored In the respective learning-time memories to a set upper limit or 

5 threshold. 

3. The neural network circuit according to Claim 1, further comprising: 

learning-time updaters (1 1 3b) for respectively subtracting a set value from the number of learning times stored 
10 In the respective learning-time memories (11 3); and 

lower limit clipping means (1 1 3d) for respectively cllpplngan subtracted result of the number of times at learning 
which is less than 0 to 0. 

4. The neural network circuit according to Claim 1 , wherein the threshold processing circuits have a function of Chang- 
's ing a threshold to a set value. 

5. A multi-layered neural network circuit provided with an input layer to which one or plural input vectors are Inputted, 
an Intermediate layer (4-11 - 4-12, 4-21 - 4-22) having neuron elements connected In tree-like structure whose 
outputs are necessarily determined by values of the input vectors and whose number corresponds to the number 

20 of the Input vectors to the Input layer, and an output layer having one or plural output neuron elements (31 0,320) 

units for Integrating all output values of said neuron elements within the intermediate layer, comprising: 

flag memories (116), respectively provided to respective paths connecting respective outputs of the interme- 
diate layer and the respective output units, for respectively memorizing the learning times or times where 
output values of neuron elements within the Intermediate layer are outputted to the corresponding path to said 
output neuron elements, when said outputs were equal to or more than a set value In the past, and for outputling 
a set value upon learning and outputting another set value in the absence of learning; and 
connection control circuits (115) for respectively controlling connection and disconnection Of the paths between 
the respective outputs of said neuron elements within the intermediate layer and respective output neuron 
elements according to an output of the respective flag memories (116), 

wherein the respective output units sum the output values of said neuron elements within the Intermediate 
layer connected by the respective connection control circuits. 

The neural network circuit according to Claim 1 or 5, wherein each output of said neuron elements within the 
Intermediate layer has two kinds of values of "HIGH" and "LOW", and the respective output neuron elements count 
the number of "HlGH"s among the outputs of said neurons within the Intermediate layer connected by respective 

connection control circuits (115). 



40 Patentanspruche 

1. VIelschlchtlge neuronale Netrschaltung mit einer EIngabeschlcht (11-11 - 12-11), In die ein oder mehrere Elnga- 
bevektoren eingegeben werden, eirier Zwlschenschlcht (11-31 - 11-38, 12-31 - 12-38), die In eIner baumartlgen 
Struktur mitelnander verbundene Neuronenelemente aufweist, deren Ausgaben notwendigenwelse durch die Wer- 
45 te der Eingabevektoren bestlmmt werden und deren Anzahl der Anzahl der EIngabevektoren der EIngabeschlcht 

entsprlcht, und einer Ausgabeschlcht mIt einem oder mehreren Ausgabe-Neuronenelementen (310,320) zum In- 
tegrleren aller Ausgabewerte der Neuronenelemente in der Zwlschenschlcht, wobei die Berechnung der Verbln- 
dung der Synapsen zu den Ausgabe-Neuronenelementen (310,32=) durch Verblndungsberechnungs-Ausfuhrtelle 
(4-1 1 - 4-22) vorgenommen wird, wobei die Verblndungsberechnungs-Ausf uhrtelle umfassen: 

so 

Lernhauflgkeltsspelcher (1 1 3), die jewells fur entsprechende Pfade, die entsprechende Ausgaben der Neu- 
ronen Ip der Zwlschenschlcht mit entsprechenden Ausgabe-Neuronenelementen verblnden, vorgesehen sind 
und die Lemhauflgkeit oder die Haufigkelt spelchem, mit der ein Ausgabewert, der von den Neuronenelemen- 

ten In der Zwlschenschlcht zu einem entsprechenden Pfad zu den Ausgabe-Neuronenelementen ausgegeben 
ss wird, groBer oder glelch einem gesetzten Wert ist, 

Schvyellwertverarbeitungsschaltungen (114), die Jewells vorgesehen sind, um die In den Lernhaufigkeltsspei- 
chern gespeicherteLernhauflgkelt mit einem gesetzten Schwellwert zu verglelchen, um einen ersten gesetzten 
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Wert auszugeben, wenn die in dem entsprechenden Lernhaufigkeitsspeichern gespeiclierte Lemhaufigkeit 
groBer oder gleich dem gesetzten Schwellwert ist, und um einen zweiten gesetzten Wert auszugeben, wenn 
die in dem entspreclienden Lemhaufigkeitsspelcher gespeicherte Lemhaufigkeit kleiner als der gesetzte 
Schwellwert Ist, und 

Verbihdungssteuerschaltungen (115), um joweils die entsprechenden Rade zwischen den entsprechenden 
Ausgaben der Neuronenelemente in der Zwischenschicht und den entsprechenden Ausgabe-Neuronenele- 
menten zu verbinden, wenn eine Ausgabe der entsprechenden Schwellwertverarbeitungsschaltungen (114) 
den ersten gesetzten Wert aufweist, und um die Rade jeweils zu unterbrechen, wenn eine Ausgabe der ent- 
sprechenden Schwellwertverarbeitungsschaltungen (114) gleich dem gesetzten Wert ist, 

wobei die entsprechenden Ausgabeeinheiten die Ausgabewerteder durch die entsprechenden \ferbindungs- 
steuerschalturtgen verbundenen Zwischenschicht summieren. . 

Neuronale Netzscha'ltung naoh Anspruch 1, die weiterhin eine Olpergrenzen-Abschneldeejnrichtung (113a) um- 
faRt, um jeweils die in den entsprechenden Lemhaufigkeitsspeichem gespeicherten Lernhaufigkeiten auf einen 
gesetzten Obergrenzwert abzusdhneiden. 

Neuronale Netzschaltung nach Anspruch 1, die weiterhin umfaBt: 

Lernhaufigkeits-Aktualisierer (.113b), Um jeweils einen gesetzten Wert von der in den entsprechenden Lern- 
° haufigkeitsspeichern (113) gespeicherten Lemhaufigkeit zu subtrahieren, und 

eine Untergrenzen-Abschnoideeinrlchtung (113d), um ein Subtraktionsergebnis der Lemhaufigkeit, das klei- 
ner ist als 0 auf 0 abzuschneiden. 

Neuronale Netzschaltung nach Anspruch 1 , wobei die Schwellwertschaltungen die Funktion haben, einen Schwell- 
wert zu einem gesetzten Wert zu andern. ' 

Vielschichtige neuronale Netzschaltung, mit einer Eingabeschicht, in die ein oder mehrere Eingabevektoren ein- 
gegeben werden, einer Zwischenschicht (4-11 -4-12, 4-21-4-22), die In einer baumartigen Struktur miteinander 
verbundene Neuronenelemente aufweist, deren Ausgaben notwendigerwelse durch die Werte der Eingabevekto- 
ren bestimmt werden und deren Anzahl der Anzahl der Eingabevektoren der Eingabeschicht entspricht, und einer 
Ausgabesohicht mit einem Oder mehreren Ausgabe-Neuronenelementen (310,320) zum Integrieren aller Ausga- 
bewerte der Neuronenelemente in der Zwischenschicht, wobei die neuronale Netzschaltung umfaRt: 

Flagspeicher.(116), die Jeweils fur entsprechende Pfade vorgesehen sind, die entsprechende Ausgaben der 
Zwischenschicht und der entsprechenden Ausgabeeinheiten verbinden, um jeweils die Lemhaufigkeit oder 
die HaufigKeit zu speichern, mit der die Neuronenelemente in der Zwischenschicht zu dem entsprechenden 
Pfad zu den Ausgabe-Neuronenelementen ausgegeben werden und die Ausgaben gr63er oder gleich einem 
zuvor gesetzten Wert sind, und um einen gesetzten Wert auszugeben, wenn gelemt wurde, und um einen 
anderen gesetzten Wert auszugeben, wenn nicht gelemt wurde, und ' 

Verblndungssteuerschaltungen (115), um jeweils die Verblndiing und Unterbrechung der Pfade zwischen den 
entsprechenden Ausgaben der Neuronenelemente InderZwIschensohiohl und den eritsprechendeh Ausgabe- 
Neuronenelementen In Ubereinstlmmung mit den entsprechenden Flagspelchern (116) zu steuern, 

wobei die entsprechenden Ausgabeeinheiten die Ausgabewerte der Neuronenelemente in der durch die ent- 
sprechenden Verblndungssteuerschaltungen verbundenen Zwischenschicht summieren. 

Neuronale Netzschaltung nach Anspruch 1 oder 5, wobei jede Ausgabe der Neuronenelemente In der Zwischen- 
schicht zwei Arten von Wertsn "HIGH" oder "LOW" aufweist, wobei die entsprechenden. Ausgabe-Neuronenele- 
mente die Anzahl der "HIGH's In den Ausgaben der Neuronenelementen In der durch die entsprechenden Steu- 
erschaltungen (115) verbundenen Zwischenschicht zahlt. 
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Circuit de r6seau neuronal multicouche muni d'une coucfie d'entr6e (11 -1 1 a 1 2-1 1 ) dans laquelle un ou plusieurs 
vecteurs d'entree sont entr6s, d'une qouche intermediaire (11-31 k 11-38, 12-31 a 12-38) ayant des elements de 
neurone connect§s en une structure du type arbre dont les sorties sont necessairement determlnees par des 
valeurs des vecteu rs d'entree et dont le nombre correspond au nombre des vecteurs d'entree de la couche d'entree, 
et d'une couche de sortie ayant un ou plusieurs elements de neurone de sortie (310, 320) pour integration de 
toutes les valeurs de sortie des elements de neurone & I'interieur de la couche intermediaire, le calcul de la con- 
nexion des synapses avec les elements de neurone de sortie (310, 320) etant execute par des parties d'execution 
de calcul de connexion (4-11 a 4-22), lesdites parties d'execution de calcul de connexion comprenant : , 

des memoires de nombre delois d'apprentissage (11 3) respectivement prevues de fapon a correspondre aux 
trajets respectifs qui connectent les sorties respectives des neurones placees k I'lntSrieur de la couche inter- 
mediaire avec les elements de neurone de sortie respectifs pour m6morlser respectivement le nombre defois 
d'apprentissage ou le nombre de fois oij une valeur de sortie des 6l6ments de neurone k I'interieur de la 
couche intermediaire qui est sortie sur un trajet correspondant vers les 6l6ments de neurone de sortie, est 
6gale ou sup6rieure a une valeur etablie ; 

des circuits de traitement de seuil (114) respectivement prevus de fapon a comparer le nombre de fols d'ap- 
prentissage memorise dans la memoire de nombre de fois d'apprentissage a ladite valeur de seuil 6tablie, 
pour sortir une premiere valeur 6tablie lorsque le nombre de fois d'apprentissage qui est memorise dans la 
m6molre de nombre de fois d'apprentissage correspondante est egal ou sup6rleur audit seuil 6tabli et pour 
sortir une seconde valeur stabile lorsque le nombre de fois d'apprentissage qui est m6moris6 dans la m6moire 
de nombre de fois d'apprentissage con-espondante est inf^rieur audit seuil etabli ; et 
des circuits de commande de connexion (115) pour connecter respectivement les trajets respectifs entre les 
sorties respectives des elements de neurone a I'interieur de la couche intermediaire et les elements de neurone 
de sortie respectifs lorsqu'une sortie des circuits de traitement de seuil correspondants (114) est ladite premiere 
valeur 6tablie et d6connecter les trajets respectifs entre ceux-ci lorsqu'une sortie des circuits de traitement 
de seuil correspondants (1 1 4) est ladite seconde valeur 6tablie, 

dans lequel les unites de sortie respectives somment les valeurs de sortie de la couche intermediaire con- 
nect§e par les circuits de commande de connexion respectifs.' 

Circuit de reseau neurprial selon la revendication 1 , comprenant de plus un moyen de fixation de limite sup6rieure 
(li3a> pour fixer respectivement le nombre de fois d'apprentissage memorise dans les mSmbires de nombre de 
fois d'apprentissage respectives a une limite ou seuil supSrieur Stabli. 

Circuit de rdseau neuronal selon la revendication 1 , comprenant de plus : 

des circuits de mise k Jour de nombre de fois d'apprentissage (113b) pour soustraire respectivement une 
valeur 6tablie du nombre de fois d'apprentissage m6moris6 dans les memoires de nombre de fois d'appren- 
tissage respectives (113) ;et 

un moyen de fixation de limite inf6rieure (113d) poUr fixer respectivement un rSsultat soustrait du nombre de 
fois d'apprehtissage.qui est inferieur a 0 a 0. 

Circuit de reseau neuronal selon la revendication 1, dans lequel les circuits de traitement de seuil ont pourfonction 

de changer un seuil a une valeur etablie. 

Circuit de reseau neuronal multicouche muni d'une couche d'entree dans laquelle un ou plusieurs vecteurs d'entree 
sont entres, d'une couche intermediaire (4-11 - 4-12, 4-21 - 4-22) ayant des elements de neurone connect6s en 
une structure de type arbre dont les sorties sont necessairement determlnees par des valeurs des yecteurs d'entree 
et dont le nombre correspond au nombre des vecteurs d'entree dans la couche d'entree, d'une couche de sortie 
ayant une ou plusieurs unites d'6l6ments de neurone (310, 320) pour int6grer toutes les valeurs de sortie desdits 
eiemerits de neurone a I'interieur de la couche intermediaire, comprenant : 

des memoires d'indicateurs (116) respectivement pr6vues sur les trajets respectifs connectant les sorties res- 
pectives de la couche intermediaire et les unites de sortie respectives, pour memoriser respectivement les 
nombres de fois d'apprentissage ou les nombres de fois ou des valeurs de sortie des elements de neurone a 
I'interieur de la couche intermediaire sont sorties sur le trajet correspondant vers lesdits elements de neurone 
de sortie, lorsque lesdites sorties etaient egales ou superieures a une valeur etablie dans le passe et pour 
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sortir une valeur etablie sur apprentissage et pour sortir une autre valeur etablle en I'absenice d'apprentissage ; 
et 

des circuits de commande de corinexion (115) pour commander respectivement la connexion et la decon- 
nexion des trajets entre les sorties respectives desdits elements de neurone a I'interieur de la couche Inter- 
m§diaire et desdits 6l§ments de neurone de sortie respectifs en conformlte avec une sortie des memolres 
d'indicateurs respectives (116), 

dans lequel les unites de sortie respectives somment les valeurs de sortie desdits elements de neurone a 
I'interieur de la couche Intermedlaire connectee par les circuits de commande de connexion respectifs. 

Circuit de reseau neuronal selon la revendication 1 ou 5, dans lequel chaque sortie desdits elements de neurone 
^ I'interieur d'une couche intermSdiaire a deux types de valeurs "HAUTE" et "BASSE" et les elements de neurone 
de sortie respectifs comptent le nombre de valeurs "HAUTES" parmi les sorties desditse neurones a I'interieur de 
la couche interm§diaire connect6s par les circuits de commande de connexion respectifs (115). 
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Fig. 8 
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